Neural Transplant Surgery: An Approach to Pre-training Recurrent Networks
نویسندگان
چکیده
Partially-recurrent networks have advantages over strictly feed-forward networks for certain spatiotemporal pattern classification or prediction tasks. However networks involving recurrent links are generally more difficult to train than their nonrecurrent counterparts. In this paper we demonstrate that the costs of training a recurrent network can be greatly reduced by initialising the network prior to training with weights ’transplanted’ from a non-recurrent architecture.
منابع مشابه
Effective Feature Selection for Pre-Cancerous Cervix Lesions Using Artificial Neural Networks
Since most common form of cervical cancer starts with pre-cancerous changes, a flawless detection of these changes becomes an important issue to prevent and treat the cervix cancer. There are 2 ways to stop this disease from developing. One way is to find and treat pre-cancers before they become true cancers, and the other is to prevent the pre-cancers in the first place. The presented approach...
متن کاملLanguage Generation with Recurrent Generative Adversarial Networks without Pre-training
Generative Adversarial Networks (GANs) have shown great promise recently in image generation. Training GANs for text generation has proven to be more difficult, because of the non-differentiable nature of generating text with recurrent neural networks. Consequently, past work has either resorted to pre-training with maximumlikelihood or used convolutional networks for generation. In this work, ...
متن کاملKnowledge Transfer Pre-training
Pre-training is crucial for learning deep neural networks. Most of existing pre-training methods train simple models (e.g., restricted Boltzmann machines) and then stack them layer by layer to form the deep structure. This layerwise pre-training has found strong theoretical foundation and broad empirical support. However, it is not easy to employ such method to pre-train models without a clear ...
متن کاملMulti-Step-Ahead Prediction of Stock Price Using a New Architecture of Neural Networks
Modelling and forecasting Stock market is a challenging task for economists and engineers since it has a dynamic structure and nonlinear characteristic. This nonlinearity affects the efficiency of the price characteristics. Using an Artificial Neural Network (ANN) is a proper way to model this nonlinearity and it has been used successfully in one-step-ahead and multi-step-ahead prediction of di...
متن کاملPre-training of Recurrent Neural Networks via Linear Autoencoders
We propose a pre-training technique for recurrent neural networks based on linear autoencoder networks for sequences, i.e. linear dynamical systems modelling the target sequences. We start by giving a closed form solution for the definition of the optimal weights of a linear autoencoder given a training set of sequences. This solution, however, is computationally very demanding, so we suggest a...
متن کامل